Appendix A: Probability distirbutions functions (PDFs) and Limit theorems
Contents
Appendix A: Probability distirbutions functions (PDFs) and Limit theorems¶
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import scipy as sp
from scipy.stats import binom, norm, poisson, expon, uniform
import holoviews as hv
hv.notebook_extension('plotly')
def get_overlay(x, pdf, cdf, points, label):
'''A function to generate overlayed interactive plotly plots showing
PDF, CDF and histogram of probability distributions'''
pdf = hv.Curve((x, pdf), label='PDF').opts(color='red')
cdf = hv.Curve((x, cdf), label='CDF', vdims='P(r)').opts(color='green')
hist = hv.Histogram(np.histogram(points, density=True), vdims='F(r)')
return (hist * pdf + cdf).relabel(label)
Uniform distribution¶
Mean
Var
PDF
CDF
hv.extension('plotly')
def uniform_dist(a, b, npts = 100, label ='Uniform Distribution'):
dist = uniform(loc=a, scale=b)
x = np.linspace(a, a+b, npts)
return get_overlay(x, dist.pdf(x), dist.cdf(x), dist.rvs(npts), label)
dmap = hv.DynamicMap(uniform_dist, kdims=['a', 'b'])
dmap.redim.range(a=(0.0,1.0), b=(2.0,10.0))
Normal dist¶
Mean
Var
PDF
CDF
hv.extension('matplotlib')
def gauss_dist(mu, sigma, npts = 100, label ='Normal Distribution'):
dist = norm(loc=mu, scale=sigma)
x = np.linspace(mu-4*sigma, mu+4*sigma, npts)
return get_overlay(x, dist.pdf(x), dist.cdf(x), dist.rvs(npts), label)
dmap = hv.DynamicMap(gauss_dist, kdims=['mu', 'sigma'])
dmap.redim.range(mu=(0.1,4.0), sigma=(0.1,10.0))
Binomial dist¶
hv.extension('plotly')
def binomial_dist(n, p, npts = 100, label ='Binomial Distribution'):
dist = binom(n, p)
x = np.arange(n+1)
return get_overlay(x, dist.pmf(x), dist.cdf(x), dist.rvs(npts), label)
dmap = hv.DynamicMap(binomial_dist, kdims=['n', 'p'])
dmap.redim.range(n=(100, 1000), p=(0.1,1))
Limit Theorems and the Laws of Large Numbers¶
Sample mean and variance¶
Consider a sequence \(X_1, X_2, \ldots\) of i.i.d. (independent identically distributed) random variables with mean \(\mu\) and variance \(\sigma^2\).
We define a partial sum or sample sum of the random variables as:
becasue of independence of random variables we have
Similarly we can the sample mean as
which has expected value and variance
Notice that the variance of the sample mean decreases to zero as n increases, implying that most of the probability distribution for \(M\) is close to the mean value.
Most importatnly we see that sample mean converges to true value with variance gowing down as \(n^{-1/2}\)
De-meaned and scaled RVs¶
We also introduce a convenient de-meaned and scaled random variable
for which
Markov Inequality¶
If a RV X can only take nonnegative values, then
Chebyshev Inequality¶
If X is a RV with mean \(\mu\) and variance \(\sigma^2\), then
An alternative form of the Chebyshev inequality is obtained by letting \(c=k\sigma\) where k is postive. This gives
which indicates that the probability of an observation of the random variable X being more than k standard deviations from the mean is less than or equal to \(1/k^2\)
Weak and Strong Law of Large Numbers¶
Weak Law: Let \(X_1, X_2, \ldots\) be i.i.d. RVs with mean \(\mu\). For every \(\epsilon > 0\)
Strong Law: The strong law of large numbers states that the sample average converges almost surely to the expected value
Convergence in Probability¶
Let \(Y_1, Y_2, \ldots\) be a sequence of RVs, not necessarily independent, and let a be a real number. We say that the sequence \(Y_n\) converges to a in probability if for every \(\epsilon \gt 0\) we have
This implies that the probability distribution of the random variables, \(Y_n\) converges to a distribution that is contained within a space of width \(2\epsilon\) around the point a. However this says nothing about the shape of the distribution.
This can be rephrased in the following way: For every \(\epsilon \gt 0\) and for any \(\delta \gt 0\), there exists \(n_0\) such that
where \(\epsilon\) is known as the accuracy and \(\delta\) is known as the confidence.
The Central Limit Theorem (CLT)¶
Let \(X_1, X_2, \ldots \) be a sequence of i.i.d. random variables with common mean \(\mu\) and variance \(\sigma^2\) snd define
Then, the PDF of \(Z_n\) converges to the standard normal PDF
Note that there is an implicit assumption that the mean and variance, \(\mu\) and \(\sigma^2\), are finite. This does not hold for certain power law distributed RVs.
Large Deviation Theorem (LDT)¶
Sum of N random variables \(X_i\) tends to a distribution which is exponentially suppressing deviaions from the mean \(Y=\frac{1}{N} \sum X_i\) with coefficient N and rate function \(I(y)\)
As N increases the LDT tends to CLT result that is gaussian distribution.
References¶
Limit Theorems and PDFs
For a smooth introduction to Prob, Stats and RV theory with lots of examples and simulation results, see:
A bit more advanced, but thorught, including conscience statement of LDT results (Chapter 5!) stochastic, inference, simulation and more in a timeless calssic,
LDT
Excellent places to start learning about utility and power of LDT in statstical mechanics are reviews and lectures given by Hugo Touchette